<html>
<head><meta charset="utf-8"><title>how to debug a deadlock · general · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/index.html">general</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/how.20to.20debug.20a.20deadlock.html">how to debug a deadlock</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="235220322"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/how%20to%20debug%20a%20deadlock/near/235220322" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Maximilian Hristache <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/how.20to.20debug.20a.20deadlock.html#235220322">(Apr 19 2021 at 17:43)</a>:</h4>
<p>Hi, I have a multithreded server which receives binary data from some clients, parse them in a dedicated thread per client then sends them to the main thread using <code>flume</code> for aggregation. </p>
<p>The server works fine under small load but seems to deadlock under heavy load.</p>
<p>I think it deadlocks as some client handling threads are stuck at 100% CPU, there is no activity on those threads visible in strace, the kernel reports bytes accumulating on the Rx side of the sockets and no change on the processed bytes if I disable the communication to the server.</p>
<p>How can I investigate where id deadlocks?</p>
<p>I tried gdb and perf but don't really understand how to interpret the data. </p>
<p>Thank you</p>



<a name="235221042"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/how%20to%20debug%20a%20deadlock/near/235221042" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/how.20to.20debug.20a.20deadlock.html#235221042">(Apr 19 2021 at 17:49)</a>:</h4>
<p>It doesn't always help, but if you attach <code>gdb</code> to the process you can see what each thread is doing. So if some threads are stuck waiting for something else, it would show up in the stack trace. But it sounds like some threads are busy processing incoming data and they never check in with the waiting threads, so it's not exactly a deadlock.</p>



<a name="235223979"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/how%20to%20debug%20a%20deadlock/near/235223979" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Maximilian Hristache <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/how.20to.20debug.20a.20deadlock.html#235223979">(Apr 19 2021 at 18:09)</a>:</h4>
<p>the thing is that there is no incoming data anylonger, the communication from the clients is disabled. The socket Rx bytes do not decrease so there is data to process but the thread does not read it it seems</p>



<a name="235224200"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/how%20to%20debug%20a%20deadlock/near/235224200" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Maximilian Hristache <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/how.20to.20debug.20a.20deadlock.html#235224200">(Apr 19 2021 at 18:11)</a>:</h4>
<p>gdb backtrace looks like this:</p>
<div class="codehilite"><pre><span></span><code>#0  0x000055906827d41d in ebmd::server::handle_ebm_stream_connection (stream=..., spec=..., tx=..., l=..., log_raw_events=false, _iias=...) at /home/bla/projects/ic/ebmd/src/server.rs:122
#1  0x000055906829d31c in ebmd::server::start_ebm_stream_instance::{{closure}} () at /home/bla/projects/ic/ebmd/src/server.rs:75
#2  std::sys_common::backtrace::__rust_begin_short_backtrace (f=...) at /home/bla/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125
#3  0x00005590682b99b1 in std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} () at /home/bla/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474
#4  &lt;std::panic::AssertUnwindSafe&lt;F&gt; as core::ops::function::FnOnce&lt;()&gt;&gt;::call_once (self=..., _args=&lt;optimized out&gt;) at /home/bla/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:344
#5  std::panicking::try::do_call (data=&lt;optimized out&gt;) at /home/bla/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379
#6  std::panicking::try (f=...) at /home/bla/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343
#7  std::panic::catch_unwind (f=...) at /home/bla/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:431
#8  std::thread::Builder::spawn_unchecked::{{closure}} () at /home/bla/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473
#9  core::ops::function::FnOnce::call_once{{vtable-shim}} () at /home/bla/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227
#10 0x00005590684a0b1a in &lt;alloc::boxed::Box&lt;F,A&gt; as core::ops::function::FnOnce&lt;Args&gt;&gt;::call_once () at /rustc/2fd73fabe469357a12c2c974c140f67e7cdd76d0/library/alloc/src/boxed.rs:1521
#11 &lt;alloc::boxed::Box&lt;F,A&gt; as core::ops::function::FnOnce&lt;Args&gt;&gt;::call_once () at /rustc/2fd73fabe469357a12c2c974c140f67e7cdd76d0/library/alloc/src/boxed.rs:1521
#12 std::sys::unix::thread::Thread::new::thread_start () at /rustc/2fd73fabe469357a12c2c974c140f67e7cdd76d0//library/std/src/sys/unix/thread.rs:71
#13 0x00007f23a2288609 in start_thread (arg=&lt;optimized out&gt;) at pthread_create.c:477
#14 0x00007f23a205a293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
</code></pre></div>



<a name="235224203"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/how%20to%20debug%20a%20deadlock/near/235224203" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/how.20to.20debug.20a.20deadlock.html#235224203">(Apr 19 2021 at 18:11)</a>:</h4>
<p>That means the thread that should process it is either:</p>
<ul>
<li>waiting on something else (and you can look at its stack trace to hopefully see on what)</li>
<li>busy doing something (and you can see in the stack trace what is it doing)</li>
</ul>



<a name="235224676"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/how%20to%20debug%20a%20deadlock/near/235224676" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Maximilian Hristache <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/how.20to.20debug.20a.20deadlock.html#235224676">(Apr 19 2021 at 18:14)</a>:</h4>
<p>the code looks like this:</p>
<div class="codehilite"><pre><span></span><code>    loop {
        match reader.fill_buf() {
            Ok(buffer) =&gt; {
                if log_raw_events {
                    info!(l, &quot;{} bytes received&quot;, buffer.len(); &quot;bytes&quot; =&gt; ?&amp;buffer[..]);
                }
                if buffer.len() == 0 {
                    info!(l, &quot;peer disconnected&quot;);
                    break;
                }
                let (n, records) = parser.parse(&amp;buffer[..]);
                consumed = n;
...
</code></pre></div>
<p>Line 122 is where the parse is called (<code> let (n, records) = parser.parse(&amp;buffer[..]);</code>)</p>



<a name="235225345"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/how%20to%20debug%20a%20deadlock/near/235225345" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Maximilian Hristache <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/how.20to.20debug.20a.20deadlock.html#235225345">(Apr 19 2021 at 18:19)</a>:</h4>
<p>so from the stack trace it looks like it's processing data but what data?</p>



<a name="235225762"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/how%20to%20debug%20a%20deadlock/near/235225762" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/how.20to.20debug.20a.20deadlock.html#235225762">(Apr 19 2021 at 18:22)</a>:</h4>
<p>Is that one of the 100% CPU threads? And I guess you're also calling <code>reader.consume(consumed)</code> below?</p>



<a name="235225937"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/how%20to%20debug%20a%20deadlock/near/235225937" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Maximilian Hristache <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/how.20to.20debug.20a.20deadlock.html#235225937">(Apr 19 2021 at 18:23)</a>:</h4>
<p>yes, and yes</p>



<a name="235226628"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/how%20to%20debug%20a%20deadlock/near/235226628" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Maximilian Hristache <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/how.20to.20debug.20a.20deadlock.html#235226628">(Apr 19 2021 at 18:28)</a>:</h4>
<p>I guess a bug could be if I return 0 as consumed from the parser</p>



<a name="235226654"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/how%20to%20debug%20a%20deadlock/near/235226654" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Maximilian Hristache <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/how.20to.20debug.20a.20deadlock.html#235226654">(Apr 19 2021 at 18:28)</a>:</h4>
<p>then the same data would be sent to the parser again I guess</p>



<a name="235227043"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/how%20to%20debug%20a%20deadlock/near/235227043" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/how.20to.20debug.20a.20deadlock.html#235227043">(Apr 19 2021 at 18:31)</a>:</h4>
<p>Yeah, you could add an <code>assert</code></p>



<a name="235227065"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/how%20to%20debug%20a%20deadlock/near/235227065" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/how.20to.20debug.20a.20deadlock.html#235227065">(Apr 19 2021 at 18:31)</a>:</h4>
<p>Actually, that might not be so unlikely</p>



<a name="235227324"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/how%20to%20debug%20a%20deadlock/near/235227324" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Maximilian Hristache <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/how.20to.20debug.20a.20deadlock.html#235227324">(Apr 19 2021 at 18:33)</a>:</h4>
<p>I will add a printout</p>



<a name="235227326"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/how%20to%20debug%20a%20deadlock/near/235227326" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> destroycomputers <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/how.20to.20debug.20a.20deadlock.html#235227326">(Apr 19 2021 at 18:33)</a>:</h4>
<p>In general that doesn't sound like a dead-lock unless you have an unbounded spinlock somewhere, normally threads would sleep in a deadlock and wouldn't generate any load on the CPU. My bet is that in the loop bit you've provided, some case isn't handled or some branch entered that makes it loop around without doing any progress. It's probably a rare case if you only can catch it under heavy load.</p>



<a name="235231712"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/how%20to%20debug%20a%20deadlock/near/235231712" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Maximilian Hristache <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/how.20to.20debug.20a.20deadlock.html#235231712">(Apr 19 2021 at 19:03)</a>:</h4>
<p>there is definitely something fishy in that loop, I added a printout and most of the time the parse gets incomplete data and returns 0. Looks like one of the clients is misbehaving. I need to dig into it more</p>
<p>Thanks <span class="user-mention" data-user-id="203546">@Laurențiu</span> and  <span class="user-mention" data-user-id="406525">@destroycomputers</span> for the help</p>



<a name="235237314"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/how%20to%20debug%20a%20deadlock/near/235237314" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Maximilian Hristache <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/how.20to.20debug.20a.20deadlock.html#235237314">(Apr 19 2021 at 19:44)</a>:</h4>
<p>I might have misunderstood how fill_buf works. I though that if I don't consume all the bytes then it will fill it up again so it will return the previously unconsumed bytes plus additional bytes read from the socket</p>
<p>But reading the docs again I am not sure if that is the case: <code>Returns the contents of the internal buffer, filling it with more data from the inner reader if it is empty.</code></p>
<p>then I wonder what is the purpose of fill_buf and consume if I need to implement additional buffering in the application </p>
<p>Any clues?</p>



<a name="235237928"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/how%20to%20debug%20a%20deadlock/near/235237928" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/how.20to.20debug.20a.20deadlock.html#235237928">(Apr 19 2021 at 19:48)</a>:</h4>
<p>I think you're only suppose to use it if you consume everything, but I might be wrong</p>



<a name="235237942"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/how%20to%20debug%20a%20deadlock/near/235237942" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Steven Fackler <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/how.20to.20debug.20a.20deadlock.html#235237942">(Apr 19 2021 at 19:48)</a>:</h4>
<p>BufReader's purpose is not to buffer an arbitrarily large number of bytes. It is to lessen the cost of many small reads.</p>



<a name="235237950"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/how%20to%20debug%20a%20deadlock/near/235237950" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Daniel Henry-Mantilla <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/how.20to.20debug.20a.20deadlock.html#235237950">(Apr 19 2021 at 19:48)</a>:</h4>
<p>Yeah, if you don't make progress (don't call <code>.consume()</code>), then <code>.fill_buf()</code> may not make any new progress. Among other things, you have to take into account that the <code>ReadBuf</code> API wants to support fixed-size buffers: if you are able to handle infinitely-sized buffers (<em>e.g.</em>, by reallocating, such as when using a <code>Vec</code> or a <code>String</code>) then you need to pull stuff out of the inner buffer and mark it as <code>.consume()</code>d, so that the <code>ReadBuf</code> knows it can overwrite that area of memory.</p>



<a name="235238275"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/how%20to%20debug%20a%20deadlock/near/235238275" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Daniel Henry-Mantilla <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/how.20to.20debug.20a.20deadlock.html#235238275">(Apr 19 2021 at 19:51)</a>:</h4>
<p>The main reason for this whole design is that the one place where unbounded buffering makes sense is for the downstream user (that is, <em>you</em>); it would be quite inefficient if the intermediate layer also performed that level of heap-allocated-and-unbounded buffering for you to potentially still have to do the same</p>



<a name="235240811"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/how%20to%20debug%20a%20deadlock/near/235240811" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Maximilian Hristache <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/how.20to.20debug.20a.20deadlock.html#235240811">(Apr 19 2021 at 20:08)</a>:</h4>
<p>thanks everyone, now I know exactly where the problem is and how to solve it! I really appreciate the help!</p>



<a name="235240844"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/how%20to%20debug%20a%20deadlock/near/235240844" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Daniel Henry-Mantilla <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/how.20to.20debug.20a.20deadlock.html#235240844">(Apr 19 2021 at 20:08)</a>:</h4>
<p>That being said, I could envision a <em>convenience</em> API being offered to help for cases such as yours. A generalization of <a href="https://doc.rust-lang.org/src/std/io/mod.rs.html#1758">https://doc.rust-lang.org/src/std/io/mod.rs.html#1758</a>:</p>
<div class="codehilite" data-code-language="Rust"><pre><span></span><code><span class="k">use</span><span class="w"> </span>::<span class="n">std</span>::<span class="n">io</span><span class="p">;</span><span class="w"></span>

<span class="k">enum</span> <span class="nc">Status</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">    </span><span class="n">Consumed</span><span class="p">(</span><span class="kt">usize</span><span class="p">),</span><span class="w"></span>
<span class="w">    </span><span class="n">Done</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>

<span class="k">fn</span> <span class="nf">collect_with</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w">    </span><span class="n">r</span>: <span class="kp">&amp;</span><span class="o">'</span><span class="nb">_</span> <span class="nc">mut</span><span class="w"> </span><span class="k">dyn</span><span class="w"> </span><span class="n">io</span>::<span class="n">BufRead</span><span class="p">,</span><span class="w"></span>
<span class="w">    </span><span class="n">f</span>: <span class="kp">&amp;</span><span class="o">'</span><span class="nb">_</span> <span class="nc">mut</span><span class="w"> </span><span class="k">dyn</span><span class="w"> </span><span class="nb">FnMut</span><span class="p">(</span><span class="o">&amp;</span><span class="p">[</span><span class="kt">u8</span><span class="p">])</span><span class="w"> </span>-&gt; <span class="nc">Status</span><span class="p">,</span><span class="w"></span>
<span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">io</span>::<span class="nb">Result</span><span class="o">&lt;</span><span class="p">()</span><span class="o">&gt;</span><span class="w"></span>
<span class="p">{</span><span class="w"></span>
<span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="k">ref</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">acc</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="fm">vec!</span><span class="p">[];</span><span class="w"></span>
<span class="w">    </span><span class="k">loop</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">        </span><span class="kd">let</span><span class="w"> </span><span class="n">r_buf</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">match</span><span class="w"> </span><span class="n">r</span><span class="p">.</span><span class="n">fill_buf</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">            </span><span class="o">|</span><span class="w"> </span><span class="nb">Ok</span><span class="p">(</span><span class="n">empty</span><span class="p">)</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">empty</span><span class="p">.</span><span class="n">is_empty</span><span class="p">()</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="k">break</span><span class="p">,</span><span class="w"></span>
<span class="w">            </span><span class="o">|</span><span class="w"> </span><span class="nb">Ok</span><span class="p">(</span><span class="n">it</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="n">it</span><span class="p">,</span><span class="w"></span>
<span class="w">            </span><span class="o">|</span><span class="w"> </span><span class="nb">Err</span><span class="p">(</span><span class="n">e</span><span class="p">)</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="fm">matches!</span><span class="p">(</span><span class="n">e</span><span class="p">.</span><span class="n">kind</span><span class="p">(),</span><span class="w"> </span><span class="n">io</span>::<span class="n">ErrorKind</span>::<span class="n">Interrupted</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="k">continue</span><span class="p">,</span><span class="w"></span>
<span class="w">            </span><span class="o">|</span><span class="w"> </span><span class="nb">Err</span><span class="p">(</span><span class="n">e</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nb">Err</span><span class="p">(</span><span class="n">e</span><span class="p">),</span><span class="w"></span>
<span class="w">        </span><span class="p">};</span><span class="w"></span>
<span class="w">        </span><span class="kd">let</span><span class="w"> </span><span class="n">r_buf_len</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">r_buf</span><span class="p">.</span><span class="n">len</span><span class="p">();</span><span class="w"></span>
<span class="w">        </span><span class="kd">let</span><span class="w"> </span><span class="n">buf</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">acc</span><span class="p">.</span><span class="n">is_empty</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">            </span><span class="c1">// try to avoid heap-allocation if caller never returns `Consumed(0)`</span>
<span class="w">            </span><span class="n">r_buf</span><span class="w"></span>
<span class="w">        </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">            </span><span class="n">acc</span><span class="p">.</span><span class="n">extend_from_slice</span><span class="p">(</span><span class="n">r_buf</span><span class="p">);</span><span class="w"></span>
<span class="w">            </span><span class="n">acc</span><span class="p">.</span><span class="n">as_slice</span><span class="p">()</span><span class="w"></span>
<span class="w">        </span><span class="p">};</span><span class="w"></span>
<span class="w">        </span><span class="k">match</span><span class="w"> </span><span class="n">f</span><span class="p">(</span><span class="n">buf</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">            </span><span class="o">|</span><span class="w"> </span><span class="n">Status</span>::<span class="n">Done</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="k">break</span><span class="p">,</span><span class="w"></span>
<span class="w">            </span><span class="o">|</span><span class="w"> </span><span class="n">Status</span>::<span class="n">Consumed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">                </span><span class="c1">// Buffer the data inside the Vec, so as to be able to `.consume()`</span>
<span class="w">                </span><span class="c1">// from the inner buffer.</span>
<span class="w">                </span><span class="k">if</span><span class="w"> </span><span class="n">acc</span><span class="p">.</span><span class="n">is_empty</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">                    </span><span class="n">acc</span><span class="p">.</span><span class="n">extend_from_slice</span><span class="p">(</span><span class="n">r_buf</span><span class="p">);</span><span class="w"></span>
<span class="w">                </span><span class="p">}</span><span class="w"></span>
<span class="w">                </span><span class="n">r</span><span class="p">.</span><span class="n">consume</span><span class="p">(</span><span class="n">r_buf_len</span><span class="p">);</span><span class="w"></span>
<span class="w">            </span><span class="p">},</span><span class="w"></span>
<span class="w">            </span><span class="o">|</span><span class="w"> </span><span class="n">Status</span>::<span class="n">Consumed</span><span class="p">(</span><span class="n">n</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">                </span><span class="k">if</span><span class="w"> </span><span class="n">acc</span><span class="p">.</span><span class="n">is_empty</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">                    </span><span class="n">r</span><span class="p">.</span><span class="n">consume</span><span class="p">(</span><span class="n">n</span><span class="p">);</span><span class="w"></span>
<span class="w">                </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">                    </span><span class="n">r</span><span class="p">.</span><span class="n">consume</span><span class="p">(</span><span class="n">r_buf_len</span><span class="p">);</span><span class="w"></span>
<span class="w">                    </span><span class="n">acc</span><span class="p">.</span><span class="n">drain</span><span class="p">(</span><span class="o">..</span><span class="w"> </span><span class="n">n</span><span class="p">);</span><span class="w"></span>
<span class="w">                </span><span class="p">}</span><span class="w"></span>
<span class="w">            </span><span class="p">},</span><span class="w"></span>
<span class="w">        </span><span class="p">}</span><span class="w"></span>
<span class="w">    </span><span class="p">}</span><span class="w"></span>
<span class="w">    </span><span class="nb">Ok</span><span class="p">(())</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<ul>
<li><a href="https://play.rust-lang.org/?version=stable&amp;mode=debug&amp;edition=2018&amp;gist=7e11752e96d9fc9483670b5b2688d8c8">Playground</a></li>
</ul>



<a name="235245646"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/how%20to%20debug%20a%20deadlock/near/235245646" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Daniel Henry-Mantilla <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/how.20to.20debug.20a.20deadlock.html#235245646">(Apr 19 2021 at 20:44)</a>:</h4>
<p>There is something funny about that <code>loop</code>: it can be rewritten as:</p>
<div class="codehilite" data-code-language="Rust"><pre><span></span><code><span class="k">loop</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">inner_consumed</span>: <span class="nc">NonZeroUsize</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="cm">/* the whole `loop` body, kind of */</span><span class="w"> </span><span class="p">};</span><span class="w"></span>
<span class="w">    </span><span class="n">r</span><span class="p">.</span><span class="n">consume</span><span class="p">(</span><span class="n">inner_consumed</span><span class="p">.</span><span class="n">get</span><span class="p">());</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>By rewriting it as such, we have a "proof" that progress is made. And in order to write it, we need to prove that <code>r_buf</code> is non-empty (thus forcing us to consider that branch, which I had forgotten to do in my first suggestion), while also necessarily forcing us to branch on the value inside <code>Consumed</code>. Pretty slick <span aria-label="nerd" class="emoji emoji-1f913" role="img" title="nerd">:nerd:</span></p>
<ul>
<li><a href="https://play.rust-lang.org/?version=stable&amp;mode=debug&amp;edition=2018&amp;gist=871009fedc7e693b42157c6c84eb52cc">Playground</a></li>
</ul>



<a name="235246578"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/how%20to%20debug%20a%20deadlock/near/235246578" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Daniel Henry-Mantilla <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/how.20to.20debug.20a.20deadlock.html#235246578">(Apr 19 2021 at 20:51)</a>:</h4>
<p><span aria-label="thinking" class="emoji emoji-1f914" role="img" title="thinking">:thinking:</span> It also makes me realize that <code>r.consume()</code> should have been made to take a <code>NonZeroUsize</code> parameter too; it would have made that part of the API clearer (cc <span class="user-mention" data-user-id="363601">@Maximilian Hristache</span>)</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>